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This report details the current progress made on, and the future expectations and 
deliverables for, NASA cooperative agreement NCC 1-401. Below is a brief summary of the 
project, followed by two sections giving the progress made on the project to date and future 

plans. 


PROJECT SUMMARY 

It is now possible - and important - to develop systems to help resolve Flight Control 
System (FCS) faults. From a human factors viewpoint, it is imperative that these systems take 
on roles, and provide functions, that are the most supportive to the pilot, given the stress, time 
pressure and workload they may experience following a FCS fault. For example, highly 
sophisticated fault recovery systems may be able to fly the aircraft following dramatic FCS 
failures without even notifying the pilot; however, such systems are not only expensive, but may 
not be able to compensate for all failures, may fail themselves, or may allow a pilot, believing he 
or she is flying a sound aircraft, to put the aircraft into a dangerous condition. Conversely, 
systems which solely provide the pilot with information about the health of the FCS may 
overwhelm the pilot with information about situations to which they could not react quickly 
enough even with a correct diagnosis. Between these two extremes, FCS fault recovery systems 
may provide several different functions, including alerting, control assistance, and decision 
aiding. 


The biggest human factors questions are in the role suitable for the technology, and its 
specific functioning to achieve that role. Specifically, for these systems to be effective, they 
must meet the fundamental requirements that (1) they alert pilots to problems early enough that 
the pilot can reasonably resolve the fault and regain control of the aircraft and that (2) if the 
aircraft’s handling qualities are severely degraded the HMS provide the appropriate stability 
augmentation to help the pilot stabilize and control the aircraft. 

This project undertook several research steps to develop such systems, focusing on the 
capabilities of pilots and on realistically attainable technologies. The ability to estimate which 
functions are the most valuable will help steer system development in the directions that can 
establish the highest safety levels. 

RESEARCH COMPLETED 

Numerical Simulations 

In the second year of performance, we developed numerical simulations of human 
performance in-the-loop with the aircraft. The underlying models of human, aircraft and flight 
control system behavior have been documented in two conference papers, both presented at the 
20 th IEEE/AI AA Digital Avionics Systems Conference in October 2002. These fault detection 
models are have been implemented in Georgia Tech’s Reconfigurable Flight Simulator (RFS) 
software, as is NASA’s model of the Boeing 737 aircraft and autoflight system dynamics. 


The first type of numerical simulation focuses on analysis of the detectability of flight 
control system faults in general, and pilot detection of faults in particular. This simulation uses 
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an extended Kalman filter, a mechanism that may serve as both a potential basis for automatic 
health monitoring systems as well as (especially when tested using a range of filter gains, 
simulating human biases and information sampling) a model of human detection performance. 
Faults are identified when the residuals of the Kalman filter exceed statistical thresholds. This 
effort therefore has several objectives: to identify the qualities of flight control system failures 
which require automated assistance to ensure reliable detection performance; to model likely 
pilot detection behavior; and to investigate the efficacy of Kalman filtering in detecting flight 
control system failures. 

The second type of numerical simulation focuses on pilot control of an aircraft 
experiencing flight control system faults and/or handling qualities degradations. This pilot 
model contains two major elements: an outer-loop control element mimicking pilot behavior in 
converting desired altitude, heading and speed into attitude and throttle commands; and an inner- 
loop element mimicking human performance at attitude tracking tasks. The objectives of this 
effort are to determine whether a fault is defected (with subsequent attempt at control by the 
pilot) so late that the aircraft is unrecoverable, and to assess whether an aircraft is damaged or 
degraded so severely that an aircraft is unrecoverable. 

Piloted Flight Simulator Experiment 

As part of this project, we conducted the second, third and fourth experiments with pilots 
in a full motion flight simulator during a day long series of tests. These experiments examined 
pilot behavior in detecting and reacting to faults in the flight control loop, with and without a 
‘Fault Meter’ and two types of Alerting Systems. In the third experiment, pilots flew multiple 
runs where the systems provided correct information; in the fourth, pilots flew a single run where 
these systems created a false alarm. 

The first experiment of the day investigated a tunnel display and several flight control 
systems, with and without simulator motion, and was intended by the collaborators to examine 
their own research issues in tunnel displays and flight simulator motion. By ‘piggy-backing’ on 
their experiment, this project had free use of a full-motion flight simulator. 

The remainder of this section summarizes the experiment and its results. 

Subjects flew SIMONA, a two-crew flight simulator with a six degree-of-freedom motion 
base. This facility is owned by Delft University of Technology (TUDelft) in The Netherlands. 

A large field-of-view, high resolution, collimated out-the-window presentation was projected 
onto a dome mounted around the simulator cab, and a detailed visual scene was created for the 
airport that included the runway, taxiways, and local features such as trees and buildings 
(described in detail at http://www.simona.tudelft.nl). 

Twelve professional pilots participated. All were recently current in jet air transport 
aircraft, ranging from regional jets to the Boeing 747-400. They had an average of 4100 flight 
hours, and all had at least 1000 flight hours in glass cockpits. 

The aircraft dynamic model simulated a small business jet, the Cessna Citation 500. No 
constant wind was used; randomly varying wind was added to the model to create moderate 
turbulence. The aircraft was flown through a control column and throttle; the pilots had no 
rudder pedals, and coordinated flight was maintained by a yaw-damper. 

The tunnel display was shown on a 15 inch LCD display in front of the subject. The 
tunnel display followed the same format as used in previous studies at TUDelft. A referenced 
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tunnel was displayed over a presentation of the outside world. Aircraft flight path was shown by 
a green flight path vector symbol. Other primary flight information was shown using common 
formats: altitude and airspeed tapes on the sides and a heading compass on the bottom. Two 
tunnel trajectories were used, which were mirrored relative to the runway centerline. The tunnel 
width was 45 meters. The tunnel trajectories were only possible to fly with a tunnel display. 

The Fault Meter and the text display of the Alerting System were displayed with the 
engine indications. The Fault Meter indicated the “lack of health” of the aircraft on a simple 
gauge from 0-100; the higher the indicated value, the worse the problem severity. A yellow arc 
indicated “caution”; a red arc, starting at 80%, indicated a “warning”. To see this instrument, 
pilots pressed a button on the control yoke, enable a record of when pilots monitored it. The 
Alerting System provided alerts based on the same information as the Fault Meter. The alerts 
had either one phase or two phases. With two-phase alerts, a distinctive sound and yellow text 
message indicated a “caution”, and a higher pitch sound and red message indicated a “warning”; 
in the one phase alerts, only warning indications were shown. 

Pilots were asked to fly curved approaches, using a tunnel display, as accurately as they 
would feel reasonable during air transport operations. After the 24 runs of the first experiment 
(conducted in Instrument Meteorological Conditions down to 200 feet altitude), pilots 
transitioned without notice to the second experiment, consisting of a single approach in which an 
unexpected Flight Control System (FCS) occurred. Pilots then had a briefing for the third 
experiment which described the Fault Meter and Alerting System, and which instructed the pilots 
to expect possible failures or problems and to use their best judgement to continue or abort the 
approach and go around. The third experiment consisted of another series of 24 approaches, this 
time in Visual Meteorological Conditions, in which faults were occasionally introduced. At the 
end, pilots were not aware they were transitioning to the fourth experiment in which a false 
alarm would be given; this last run was otherwise identical to those before. 

So that multiple faults could be tested without pilots experiencing the same fault twice, 

12 faults were implemented. These faults were calibrated in severity such that tunnel tracking 
would be difficult but not impossible, especially if the pilot was able to recognize the fault and 
develop an appropriate coping strategy. The faults were triggered at varying points during the 
approach to limit their predictability. Six faults were specific to the direct-link FCS and six to the 
flight-path-oriented FCS, as their impact on the pilots’ tasks depended on FCS. 

Each run started with the aircraft positioned in a 15 km long tunnel to the runway, 
properly configured for the approach with the autothrottle engaged. A yaw-damper established 
coordinated flight; this reduced the pilot control task to controlling pitch and roll through the 
control yoke. Pilots were asked to disconnect the autothrottle upon reaching 500 feet AGL to 
reduce speed for final approach and landing. An experimenter acted as first officer, but did not 
perform any flying tasks; he did call out radar altitudes as per standard operating procedures. 
After each run the pilot provided a NASA Task Load Index (TLX) workload assessment and 
opinions on the fault, Fault Meter and Alerting Systems. 

Twelve faults were implemented. These faults were calibrated in severity such that 
tunnel tracking would be difficult but not impossible, especially if the pilot was able to recognize 
the fault and develop an appropriate coping strategy. The faults were triggered at varying points 
during the approach to limit their predictability. 

A total of 312 data runs were recorded in these experiments; of these, 144 were in the 
third experiment, 12 were with the unexpected fault in the second experiment, and 12 were false 
alarm conditions. 
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Pilots were able to give at least a minimally correct assessment of what had occurred in 
89% of the runs, and in nearly half the runs attempted to diagnose the cause of fault; half their 
diagnoses were correct (27% of the total runs). However, 1 1% of the time pilots did not know, 
or gave an incorrect description of, both symptoms and cause. 

Pilots were asked to describe the fault at the end of every run. In 16 of the runs (1 1%), 
the pilots either provided a wrong guess or did not know what had happened. In the remaining 
cases, the pilots correctly identified some aspect of the fault, ranging from an ‘Acceptable 
Comment’ with only awareness of general attributes of the fault to attempts to diagnose 
(correctly or not) the root cause. 

Pilots were asked to rate the alerting system and fault meter overall in a questionnaire at 
the end of the experiment. No consistent trend was found: nine pilots ranked the Alerting 
System as more useful than the Fault Meter and the remaining three gave the opposite rating. To 
see the Fault Meter, the pilot had to press a button located on the yoke, providing a measure of 
the use of the instrument. Analysis of this measure shows that two of the three pilots that ranked 
the Fault Meter higher than the alerting system in the questionnaires also scanned the instrument 
at a much higher frequency than pilots that preferred the alerting system. This behavior is also 
consistent with their comments indicating that they “scanned the instrument frequently”. This 
type of behavior may suggest that a periodic scan of the instrument was established to take 
advantage of the initial trend of the indicator, seeking an earlier indication of problems. 

At the end of each run, pilots also described their workload using the 0-100 NASA TLX 
workload rating scales. ANOVA found the two FCS had effects on all measures except for 
Frustration (p<0.05). Fault Meter and Alerting System did not have any effect on the TLX 
measures. “Frustration” was found to diminish by Run Order (p<0.05). 

Tunnel tracking error increased significantly after the fault (F=73.82, p<0.001). This 
error was the highest after the fault when the pilot immediately declared a go-around, and next 
highest when the pilot either eventually declared a go-around or crashed; these differences were 
found to be significantly different by Tukey multiple comparisons at the level of p<0.05. The 
error in the ‘Immediate Go Around’ cases may be slightly inflated by the high tracking error at 
the moment when pilots abandoned the tunnel for a go around. 

The pilot’s descriptions of the false alarm condition in the final fourth experiment 
indicated that over half did believe that something was wrong with the aircraft, suggesting that 
pilot’s will be vulnerable to false alarms from health monitoring systems. Likewise, an ANOVA 
found a marginally significant difference in tunnel tracking before and after the false alarm in the 
tracking performance (p=0.1 for 10 data points). 

The wide variety of pilot responses suggest that both training and cockpit systems can be 
developed to help pilots detect and respond to faults in the flight control loop. To date, training 
and design interventions have focused on helping pilots in response to specific faults; for 
example, diagnostic systems center on identifying problems with specific aspects of the aircraft 
or specific problems. While these efforts are valuable, the cost of sensing every aspect of the 
aircraft can be prohibitive, limiting the broader applicability of these systems; likewise, not every 
conceivable problem can be prepared for advance. As such, systems capable of detecting 
unusual aircraft behavior without giving specific diagnoses, such as the fault meter and alerting 
system tested here, may be worthy of investigation if effective methods can be communicated to 
pilots for incorporating its information into diagnosis and pilot decision making. 
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